Power Law and Entropy

Shannon[1] and Yavuz[2] estimated the entropy of real documents. This note drives an upper bound of entropy from the power law. Let D be a set of documents and f r(w) be the number of occurrences of a word w in the document. Given an interger k, we denote by F (k) the number of words whose frequency is k, i.e., F (k) = ♯{w ∈ D | f r(w) = k}. The entropy of the document set D is defined by E(D) = Σ w∈D p(w)log(p(w)), where p(w) = f r(w)/N and N = Σ w∈D f r(w).